Lightweight Spatial-Channel Adaptive Coordination of Multilevel Refinement Enhancement Network for Image Reconstruction

Benefiting from the vigorous development of deep learning, many CNN-based image super-resolution methods have emerged and achieved better results than traditional algorithms. However, it is difficult for most algorithms to adaptively adjust the spatial region and channel features at the same time, let alone the information exchange between them. In addition, the exchange of information between attention modules is even less visible to researchers. To solve these problems, we put forward a lightweight spatial-channel adaptive coordination of multilevel refinement enhancement networks(MREN). Specifically, we construct a space-channel adaptive coordination block, which enables the network to learn the spatial region and channel feature information of interest under different receptive fields. In addition, the information of the corresponding feature processing level between the spatial part and the channel part is exchanged with the help of jump connection to achieve the coordination between the two. We establish a communication bridge between attention modules through a simple linear combination operation, so as to more accurately and continuously guide the network to pay attention to the information of interest. Extensive experiments on several standard test sets have shown that our MREN achieves superior performance over other advanced algorithms with a very small number of parameters and very low computational complexity

Image Reconstruction of Multi Branch Feature Multiplexing Fusion Network with Mixed Multi-layer Attention

Image super-resolution reconstruction achieves better results than traditional methods with the help of the powerful nonlinear representation ability of convolution neural network. However, some existing algorithms also have some problems, such as insufficient utilization of phased features, ignoring the importance of early phased feature fusion to improve network performance, and the inability of the network to pay more attention to high-frequency information in the reconstruction process. To solve these problems, we propose a multi-branch feature multiplexing fusion network with mixed multi-layer attention (MBMFN), which realizes the multiple utilization of features and the multistage fusion of different levels of features. To further improve the networks performance, we propose a lightweight enhanced residual channel attention (LERCA), which can not only effectively avoid the loss of channel information but also make the network pay more attention to the key channel information and benefit from it. Finally, the attention mechanism is introduced into the reconstruction process to strengthen the restoration of edge texture and other details. A large number of experiments on several benchmark sets show that, compared with other advanced reconstruction algorithms, our algorithm produces highly competitive objective indicators and restores more image detail texture information

RMFNet: Redetection Multimodal Fusion Network for RGBT Tracking

The development of single-modality target tracking based on visible light has been limited in recent years because visual light images are highly susceptible to environmental and lighting influences. Thermal infrared images can well compensate for this defect, so RGBT tracking has attracted increasing attention. However, existing studies are limited to the aggregation of multimodal information using feature fusion, ignoring the role of decision-level fusion in tracking, and the original re-detection algorithm in the used baseline model is prone to the accumulation of failures. To deal with these problems, we propose the Redetection Multimodal Fusion Network (RMFNet). The network is divided into three branches, the visible light branch, the thermal infrared branch, and the fusion branch. The three-branch structure can plainly utilize the complementary advantages of multimodal information and the commonalities and specific characteristics of the two modalities. We propose a multimodal feature fusion module (EFM), which can adaptively calculate the reliability of the modality and perform a weighted fusion of the two-modality features features. The existing redetection algorithm is improved, and the re-detection mechanism of global search in the current frame is added to reduce the accumulation of failures. We have conducted extensive comparative validation on two widely used benchmark datasets, GTOT and RGBT234. The outcomes of the experiments suggest that RMFNet outperforms other tracking methods

Optical Flow-Aware-Based Multi-Modal Fusion Network for Violence Detection

Violence detection aims to locate violent content in video frames. Improving the accuracy of violence detection is of great importance for security. However, the current methods do not make full use of the multi-modal vision and audio information, which affects the accuracy of violence detection. We found that the violence detection accuracy of different kinds of videos is related to the change of optical flow. With this in mind, we propose an optical flow-aware-based multi-modal fusion network (OAMFN) for violence detection. Specifically, we use three different fusion strategies to fully integrate multi-modal features. First, the main branch concatenates RGB features and audio features and the optical flow branch concatenates optical flow features with RGB features and audio features, respectively. Then, the cross-modal information fusion module integrates the features of different combinations and applies weights to them to capture cross-modal information in audio and video. After that, the channel attention module extracts valuable information by weighting the integration features. Furthermore, an optical flow-aware-based score fusion strategy is introduced to fuse features of different modalities from two branches. Compared with methods on the XD-Violence dataset, our multi-modal fusion network yields APs that are 83.09% and 1.4% higher than those of the state-of-the-art methods in offline detection, and 78.09% and 4.42% higher than those of the state-of-the-art methods in online detection

Image Reconstruction of Multibranch Feature Multiplexing Fusion Network with Mixed Multilayer Attention

Image super-resolution reconstruction achieves better results than traditional methods with the help of the powerful nonlinear representation ability of convolution neural network. However, some existing algorithms also have some problems, such as insufficient utilization of phased features, ignoring the importance of early phased feature fusion to improve network performance, and the inability of the network to pay more attention to high-frequency information in the reconstruction process. To solve these problems, we propose a multibranch feature multiplexing fusion network with mixed multilayer attention (MBMFN), which realizes the multiple utilization of features and the multistage fusion of different levels of features. To further improve the network’s performance, we propose a lightweight enhanced residual channel attention (LERCA), which can not only effectively avoid the loss of channel information but also make the network pay more attention to the key channel information and benefit from it. Finally, the attention mechanism is introduced into the reconstruction process to strengthen the restoration of edge texture and other details. A large number of experiments on several benchmark sets show that, compared with other advanced reconstruction algorithms, our algorithm produces highly competitive objective indicators and restores more image detail texture information

Sand Dust Images Enhancement Based on Red and Blue Channels

The scattering and absorption of light results in the degradation of image in sandstorm scenes, it is vulnerable to issues such as color casting, low contrast and lost details, resulting in poor visual quality. In such circumstances, traditional image restoration methods cannot fully restore images owing to the persistence of color casting problems and the poor estimation of scene transmission maps and atmospheric light. To effectively correct color casting and enhance visibility for such sand dust images, we proposed a sand dust image enhancement algorithm using the red and blue channels, which consists of two modules: the red channel-based correction function (RCC) and blue channel-based dust particle removal (BDPR), the RCC module is used to correct color casting errors, and the BDPR module removes sand dust particles. After the dust image is processed by these two modules, a clear and visible image can be produced. The experimental results were analyzed qualitatively and quantitatively, and the results show that this method can significantly improve the image quality under sandstorm weather and outperform the state-of-the-art restoration algorithms